The effects of collocational strength and contextual predictability in lexical production

0 Introduction Word frequency and word predictability have both been proposed in the literature as explanations for word shortening or reduction. Traditionally, these two explanations have been modeled separately. Frequency models focus on the fact that words with high use frequency are shortened compared to low frequency words, whether in the lexicon (Zipf 1929) or during phonetic production (Fidelholtz 1975, Bybee 1999a). Predictability models focus on the fact that words that are highly predictable from the context are shortened during production (Jespersen 1922, Bolinger 1981, Fowler & Housum 1987). We propose that these “predictability” and “frequency” affects are actually variants of the same basic factor: the informativeness of a word as measured by its probability. In this account, words which are highly predictable or very frequent are highly probable, and hence have a lower information value. A consequence of considering frequency and predictability as probabilities is that they can be unified into a probabilistic model of processing together with other types of probabilistic knowledge, ultimately providing a more complete explanation of language use. Probabilistic models of human language comprehension claim that probabilistic information about words, phrases, and other linguistic structure is represented in the minds of language users and plays a role in language comprehension (Jurafsky 1996, Narayanan & Jurafsky 1998). This paper extends this probabilistic hypothesis to language production, suggesting that speakers use their knowledge of the probability of a word or combinations of words in sentence production. In particular, we present evidence that highly probable (less informative) words are shorter or more reduced in conversational speech. This is true whether the high probability of the word is based on frequency, collocation with neighboring words, repetition of the word in the conversation, or the semantic association of the word with its conversation context. Our results are based on experiments designed to test probabilistic effects on three measures of shortening: deletion, tapping, and durational shortening. The effects of various probabilistic measures were tested on 8,472 word tokens ending in t or d taken from the phonetically transcribed Switchboard corpus of American telephone conversations (Greenberg et al. 1996). In §1 we review the range of probabilistic measures known to affect production and argue that frequency and predictability are probabilistic measures. We describe the methodology of our study in §2, defining the probabilistic measures considered as predictors in §2.3. The results of the models are presented in §3, with a discussion following in §4. In §5, we discuss the implications of our results for probabilistic models of production.